Robust video retrieval utilizing video data

ABSTRACT

Techniques for determining if two video signals match by extracting features from a first and second video signal, and cross-correlating the features thereby providing a cross-correlation score at each of a number of time lags, and finally determining the similarity score based on both the cross-correlation scores.

RELATED APPLICATION(S)

This application is a continuation-in-part of U.S. application Ser. No.12/538,476, filed Aug. 10, 2009.

The entire teachings of the above application(s) are incorporated hereinby reference.

BACKGROUND

1. Field

The present application relates generally to digital media and morespecifically to the process of quickly, efficiently and accuratelyretrieval similar videos based on extracted feature comparison.

2. Description of the Related Art

Regarding content based video retrieval, one of two main approaches areusually employed. The first is related to matching specific extractedkey frames from one video to another. Key frames are extracted atregular intervals, or sometimes selected by scene change detectionalgorithms. A popular approach is to simply compare key frames of videosusing new or existing content-based image retrieval (CBIR). The secondis related to modeling the entire clip, and performing a model basedcomparison during the retrieval. Another existing video retrievaltechnique is to model entire video clips in some manner, and thenperform a model comparison during retrieval. While other models areavailable, the main model used is a temporal model.

Key Frame Comparison

Key frames are often extracted at regular intervals, or sometimesselected by scene change detection algorithms. A popular approach is tosimply compare key frames of videos using new or existing content-basedimage retrieval (CBIR). However, this analysis suffers from two largeshortcomings.

Some specific examples of existing technology that utilizes key framecomparison for video retrieval are as follows.

1) U.S. Pat. No. 5,982,979

The video retrieving method provides a video retrieval man-machineinterface which visually specifies a desired video out of many storedvideos by using previously linked picture data corresponding to thevideos. Also, a video reproduction operating man-machine interfacevisually designates the position of reproduction out of the picturegroup indicative of the contents. The video retrieving method employsvideo data, character information linked to the video data, picture datalinked to the videos, and time information corresponding to the picturedata in the video data. The character information is composed of a titleof each video and a creation date thereof. The picture data include, asretrieval information, one picture data representing the content of therelevant video (one picture expressing the video, i.e., a leaflet or thelike), and a plurality of picture data adapted to grasp the contents ofthe entire video. The time information indicates the temporal positionof the picture data in the video data.

Hauptmann, A. G., Christel, M. G., and Papernick, N. D., Video Retrievalwith Multiple Image Search Strategies, Joint Conference on DigitalLibraries (JCDL, '02), Portland, Oreg., pp. 376, Jul. 13-17, 2002describes the Informedia digital video library which provides automaticanalysis of video streams, as well as interactive display and retrievalmechanisms for video data through various multimedia surrogatesincluding titles, storyboards, and skims.

Another existing video retrieval technique is to model entire videoclips in some manner, and then perform a model comparison duringretrieval. While other models are available, the main model used is atemporal model.

One example of existing technology that utilizes temporal modeling forvideo retrievals is in Chen, L. and Stentiforda, F. W. M., Videosequence matching based on temporal ordinal measurement, PatternRecognition Letters, Volume 29, Issue 13, 1 Oct. 2008, Pages 1824-1831.That paper proposes a novel video sequence matching method based ontemporal ordinal measurements. Each frame is divided into a is grid andcorresponding grids along a time series are sorted in an ordinal rankingsequence, which gives a global and local description of temporalvariation. A video sequence matching means not only finding which videoa query belongs to, but also a precise temporal localization. Robustnessand discriminability are two important issues of video sequencematching. A quantitative method is also presented to measure therobustness and discriminability attributes of the matching methods.Experiments are conducted on a BBC open news archive with a comparisonof several methods.

Another approach using temporal modeling is described in Chen, L., Chin,K. and Liao, H., An integrated approach to video retrieval, ACMInternational Conference Proceeding Series Vol. 313, Proceedings of thenineteenth conference on Australasian database—Volume 75, 2008, Pages49-55. There it is described that the usefulness of a video databasedepends on whether the video of interest can be easily located. Thispaper proposes a video retrieval algorithm based on the integration ofseveral visual cues. In contrast to key-frame based representation ofshot, the approach analyzes all frames within a shot to construct acompact representation of video shot. In the video matching step, byintegrating the color and motion features, a similarity measure isdefined to locate the occurrence of similar video clips in the database.

U.S. Pat. No. 7,486,827 describes a two-step matching technique isembodied in a video-copy-detection algorithm that detects copies ofvideo sequences. The two-step matching technique uses ordinal signaturesof frame partitions and their differences from partition mean values.The algorithm is not only robust to intensity/color variations it canalso effectively handle various format conversions, thereby providingrobustness regardless of the video dynamics of the frame shots.

BRIEF SUMMARY

These prior art approaches each have limitations.

With the key frame approach:

1) If frames are extracted based on a non-temporal basis (i.e., a setnumber of frames are skipped between each key frame), then differencesin frames per second (fps) will cause extracted key frames from similarvideos to not align properly, yielding inaccurate results. Furthermore,if the temporal alignment is very close, but not 100% the same, it ispossible for a scene change to cause a different frame to be selectedfor two videos at the same point in time.

2) If a scene change detection algorithm is used to select key frames,the retrieval will only be as good as the scene change detectionalgorithm, and a propagation of error may be in effect. From experience,it is very rare to see the exact same set of key frames extracted fromtwo videos which were encoded differently from the same source. Whilethe sets of extracted key frames need not be identical, variationsultimately impact relevance ranking in the matched results.

Temporal modeling often suffers from three large shortcomings asfollows.

1) Aligning video data in the time domain is not an easy task. Reliablydetermining an exact frames per second (fps) value, and then extractingframes uniformly based on that fps becomes a large dimension to theretrieval problem.

2) Videos with little to no motion are very difficult to model in thetime domain.

3) Very short videos often yield very little data to be temporallymodeled, regardless of content.

To overcome these and other shortfalls, a system and/or method for videocomparison can determine if two video signals match by first extractingfeatures from a first and second video signal, cross-correlating theextracted features thereby providing a cross-correlation score at eachof a number of time lags; and then outputting an indication of a degreeof match between the first and second video signals.

The target video signal is considered to be a candidate match againstthe query video signal when the similarity score is over the comparisonthreshold, it may also determine that target video signal matches thequery video signal at an interval corresponding to the lag resulting ina highest combined score.

Furthermore, the feature is extracted from each frame the video signalsmay be based on a mean value of grayscale pixel values in the frame.

In another configuration, the system and/or method may determine if eachof the feature-extracted signals is substantially constant over time,and if not, then determining if there is a match from individual framespatial features extracted from frames, and tested at regular intervalsover the match period.

The above discussion of systems and/or methods is meant to be but abrief overview, and not a complete or sufficient description of thepresent invention, which should be considered as being defined by theclaims that follow.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIG. 1 illustrates the general process flow and decision process for thevideo extraction.

FIGS. 2( a) and 2(b) show an example of a letterbox introduced issue.FIG. 2( a) was extracted from the original, 16:9 version of the videostream. FIG. 2( b) was extracted from a version of the video where theoriginal 16:9 stream was re-encoded into a 4:3 format with a severeblack border resulting on the top and bottom, and minor black bordersresulting on the left and right.

FIG. 3( a) represents the horizontal scan line edge variance and FIG. 3(b) represents the vertical scan line edge variance for FIG. 2( a). FIG.3( a) and FIG. 3( b) represent the horizontal scan line edge varianceand vertical scan line edge variance for FIG. 2( b), respectively. Thereis no area to remove from FIG. 3( a) and FIG. 3( b), and the area toremove from FIG. 3( c) and FIG. 3( d) is indicated by the shaded,outlined regions 300, 301, 302 and 303.

FIGS. 4( a) and 4(b) show the result of the letterbox cropping for thetwo frames shown in FIGS. 2( a) and 2(b). FIG. 4( a) was not cropped atall, whereas a significant amount was cropped from FIG. 4( b).

FIG. 5 illustrates an example of one time series signal derived from avideo stream.

FIG. 6 illustrates an example flatline signal.

FIG. 7 illustrates the spatial frame edge map feature, and how it isquantized and stored in binary format.

FIG. 8 illustrates the local histogram equalized luminance feature, andhow it is quantized and stored in binary format.

FIG. 9 illustrates the global frame color histogram feature, and how itis quantized and stored in binary format.

FIG. 10 illustrates the general process flow and decision process forthe audio extraction.

FIG. 11 shows an example of the audio feature signal.

FIG. 12 illustrates the general process flow and decision logic for thevideo matching, audio matching, and combined video and audio matchingalgorithm

FIG. 13 illustrates the video matching technology housed within thecompute environment for which it was designed.

FIG. 14-FIG. 16 show select result cases from the technology inoperation.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

Introduction

With the increasing presence of online multimedia as well as the widegamut of formats and encoding schemes available, the need for accurateand efficient video retrieval schemes are of growing importance. Videosimilarity matching is trivial when both the query and target videogenerate the same md5 hash value. Additionally, it is not very difficultto compare files in the same video format. In addition, metadata thatcan be extracted from multimedia files can also make searching a trivialtask, as the algorithm becomes an exercise in text comparison. However,when md5 values and metadata either do not exist or are not helpful indetermining matching, the only other data to use is derived from themultimedia content itself.

The present system, a Content-based Video Retrieval (CBVR) system,analyzes both video and audio streams from digital multimedia files, andcalculates a feature (or features) that uniquely and abstractlydescribes the media. These features are then compared against each otherduring the retrieval process in order to establish their “similarity”.Unfortunately, several factors can contribute to inaccuracies during theretrieval process, including, but not limited to, changes in fileformats and encoding procedures, changes in content quality, and absentvideo and/or audio data. The possible factors that contribute toretrieval inaccuracy are discussed in more detail below. These factorsexist naturally and frequently in real world applications, so it isimportant to select features that are invariant to these types ofirregularities. The present system thus emphasizes tolerance to thevariability found in a large generalized corpus of audio and videofiles.

Problem Statement

Given an arbitrary query multimedia file that contains video content,audio content, or both, find other similar multimedia files in a largerepository of files in a quick and efficient manner. Similarity, in thiscontext, is defined as multimedia files that may or may not have md5hash duplicates of each other, do not have matching metadata, but havethe same video/audio content differing by one or more of the followingcharacteristics:

-   -   The query video or audio data is a subclip of the target video        (or vice versa)    -   The query video and target video differ by quality or integrity        (data corruption)    -   The query video and target video are of differing file and        encoding formats    -   The query video and target video contain little to no temporal        variation (content does not change much over time, e.g. a        stationary surveillance video)    -   The query video and target video differ by frame size    -   The query video and target video differ by aspect ratio    -   The query video and target video differ by color saturation    -   The query video and target video differ by contrast and/or        brightness    -   The query video and target video contain no audio information    -   The query asset and target asset contain no video information

Current Technical Approach of the Preferred Embodiment

The preferred multimedia retrieval approach utilizes features from bothdigital video and audio data. In a specific example, digital video, aswell as digital audio (such as in the form of extracted dual channel22050 Khz Pulse Code Modulated (PCM) data) are used. They are firstcorrectly aligned in the time domain. Features are then extracted fromthe data and temporal signatures are created.

In addition to a temporal signature, specific spatial features arecomputed for each frame. A combination of cross correlation analysis (inthe case of temporal features) and direct bit-wise comparison (in thecase of spatial frame features) are then used during the retrievalprocess to match extracted features of digital media to one another.

Feature Extraction

For each media asset, an attempt to extract both audio and videofeatures takes place. If either the audio or video stream is unavailableor encoded with an unrecognized codec, no feature extraction is possiblefor that stream. In addition, if the extracted stream yields 0 frames,no feature extraction is attempted for that stream. In other words, atleast 1 frame must be extracted from the video for visual featureextraction to take place.

Video Feature Extraction

FIG. 1 illustrates the general process flow and decision process for thevideo extraction. A very brief summary of the process is as follows: Instep 100, a new, incoming video file is presented to the system. In step101, an attempt is made to extract frames from the video and a decisionis made depending on the outcome. If at least 1 frame was not able to beextracted, in step 102 the process is terminated with no video featurecreated. Otherwise, step 103 performs a letterbox cropping filter oneach extracted frame. Step 104 forces each frame to be resized to318×240. Step 105 extracts a single, statistical measurement from eachframe. Step 106 computes 3 separate spatial features for each frame. Instep 107, the single frame measurement is converted into temporalsignals. In step 108, a flatline analysis is performed on the globalframe temporal signal. In step 109, the video featuring is completelydone.

The following is a more detailed summary of the step-by-step process forvideo feature extraction:

-   -   1) In step 101, JPG frames are extracted from the original media        file at a rate of 1 frame every 500 ms (or 2 frames per second).        If at least one frame cannot be extracted, the process is        aborted and no feature is created.    -   2) For each frame extracted during step 101, a letterbox        cropping filter is applied (step 103) to the frame to remove a        possible border artifact. This is performed on frames that        contain a black border and will increase the likelihood of        matching the frame with a black border to the original video        because both frames have different aspect ratios. FIG. 2 shows        an example of a letterbox introduced issue. FIG. 2 a was        extracted from the original, 16:9 version of the video stream.        FIG. 2 b was extracted from a version of the video where the        original 16:9 stream was re-encoded into a 4:3 format with a        severe black border resulting on the top and bottom, and minor        black borders resulting on the left and right.

The Letterbox cropping filter first performs an edge analysis on theframe. There are many edge detection algorithms available, for example,the Sobel edge operator described in Sobel, I., Feldman, G., “A 3×3Isotropic Gradient Operator for Image Processing”, presented at a talkat the Stanford Artificial Project in 1968, unpublished but often cited,orig. in Pattern Classification and Scene Analysis, Duda, R. and Hart,P., John Wiley and Sons, '73, pp 271-2, and the Canny edge detectordescribed in Canny, J., A Computational Approach To Edge Detection, IEEETrans. Pattern Analysis and Machine Intelligence, 8:679-714, 1986.

However, for speed and efficiency, a simple edge detection algorithmbased on local pixel variance over a threshold can be used. With thisapproach, a horizontal and vertical scan line variance analysis of theresulting edge map is conducted. Continuous regions of low scan linevariance measured from the edge of the frame are discounted from featureconsideration. A scan line standard deviation of 0.05 is used as athreshold. This value performs well at detecting actual edge in videoframes, while suppressing possible minor false edges that may be anartifact of JPG compression.

In FIG. 3, panels a and b represent the horizontal scan line edgevariance (panel a) and vertical scan line edge variance (panel b) forFIG. 2 a. Panels c and d in FIG. 3 represent the horizontal scan lineedge variance (panel c) and vertical scan line edge variance (panel d)for FIG. 2 b. There is no area to remove from FIGS. 3 a and 3 b, and thearea to remove from FIGS. 3 c and 3 d is indicated by the shaded,outlined regions 300, 301, 302 and 303. FIG. 4 shows the result of theletterbox cropping for the two frames shown in FIG. 2. FIG. 4 a was notcropped at all, whereas a significant amount was cropped from FIG. 4 b.

Because the intended use of the filter was to remove possible borders,some rules have been incorporated to prevent too much cropping frominadvertently happening. The first rule is that if the total left andright combined removal amount is more than 35% of the total frame width,the frame is untouched. Similarly, the second rule is that if the totaltop and bottom combined removal amount is more than 35% of the totalframe height, the frame is untouched. These amounts were derived bytaking each expected worst possible case, a 16:9 forced into a 4:3resulting in black top/bottom borders (rule 2), and a 4:3 forced into a16:9 resulting in black left/right borders (rule 1), plus a little bitextra for expected error.

-   -   3) In step 104, the resulting letterbox filtered image then        undergoes a forced resize to 318×240 for two reasons: 1) The        computed features are very spatially generic, and do not suffer        from reduced quality or lack of fidelity at the local pixel        level, and 2) Reducing large frame sizes (sometimes as high as        1000×500 and larger) greatly reduces feature extraction and        further processing time.    -   4) From step 104, once each frame has been resized, then the        actual feature extraction takes place. The global frame feature        in step 105 is a single point measurement that uniquely        describes the frame that we are analyzing, so a measurement was        picked that incorporated global pixel averages and global pixel        standard deviations.

-   Mean value of frame pixel grayscale values divided by 255.0

-   Stdev value of frame pixel grayscale values divided by 127.5

The denominators in each calculation normalize each value to [0.0, 1.0].The goal of this feature extraction measurement is to represent thedistribution of target pixel values as simply and as descriptively aspossible by a single, scalar value. As statistical distributions aredescribed by their moments, the two components (stdev and mean,respectively) represent the first two moments of the distribution.However, while both features are being extracted (mean, stdev),currently only the mean value is being used during matching, as thissupplies sufficient discrimination for an initial pass.

-   -   5) At step 107, when the preceding has been done for all frames        extracted for the video, this yields 2 time series signals (time        vs. measurement). In all, there are two signals derived from one        video stream, the signal for the 2 global frame measurements.        FIG. 5 illustrates an example of one time series signal derived        from a video stream.    -   6) As will be described below, the retrieval process relies on        differences and uniqueness in these derived features. However,        as is sometimes the case, video streams do not have a lot of        signal change, especially short videos. For example, a 15 second        clip captured from a surveillance camera showing little to no        activity (as detected in step 108) would have little to no        change in frame to frame point measurements, thus resulting in a        “flatline” signal. See FIG. 6 for an

A signal is identified as being a flatline signal based on both thestatistical distribution of the temporal data points as well as thelength of the video. The longer the video and the more varied thetemporal point data is, the greater the chance that a video will not betagged as a flatline.

-   Component 1=mean value of calculated global frame means (temporal    feature 1)-   Component 2=mean value of calculated global frame stdevs (temporal    feature 2)

Specifically, a video is tagged as flatline when the following conditionis met:

1.0>[(numFrames*Component 1)+(numFrames*Component 2)]/2.0

Flatline analyses on video streams are used and are useful in aidinglogic and branching path decision during the matching process.

In addition to a single point measurement being calculated on the globalframe, a set of 3 features are being extracted from each frame:localized detected edge, localized luminance-based feature, and a globalcolor histogram in step 106 on the frame. The specific spatial framefeatures are described in detail below:

Localized Edge Detection:

Using the same edge detection algorithm used in the letterbox detectiontechnique outlined above, edge is detected on the frame and an edge mapof the frame is created. The edge map is quantized into a 16×16 grid,and the edge map grid is then encoded into a binary matrix of 0's and1's, and stored numerically until comparison. FIG. 7 shows an example ofthe edge map feature, and how the edge map is quantized into a grid, andeach cell is encoded into a binary map and stored numerically

Localized Luminance:

The frame is converted to grayscale, and then a histogram equalizationis computed to correct for variations in luminance and contrast. Thehistogram equalized grey values are then quantized into a 16×16 grid,and the grid is encoded in binary such that the upper 50% of values areencoded as a 1, and the bottom 50% of values are encoded as a 0. Thematrix is then stored numerically until comparison. FIG. 8 shows anexample of the histogram equalized luminance feature, and how theluminance is quantized into a grid, and each cell is encoded into abinary map and stored numerically.

Global Color Histogram

A 16-color global color histogram is computed on each frame. The framescolors are quantized into 16 bins based on closest proximity in 3D RGBspace, a histogram is created based on the frequency of each of the 16quantized colors, and finally this histogram is represented in binary,and is stored numerically until comparison. FIG. 9 shows an example ofthe global color histogram feature, and how the color is encoded into abinary map and stored numerically.

“Blank” Frames:

An issue with these frame features, how they are computed, and how theyare compared to each other is the notion that blank or near blank frames(i.e., a blank frame with a single white word across the center),actually compare very, very highly, even though the actual content ofthe frame is completely dissimilar. We have developed a concept of a“blank” frame, or more specifically, these 3 frame spatial features areonly computed if there is deemed enough valid frame content to perform ameaningful frame comparison. A frame is determined to be “nonblank” ifthe global standard deviation of its grayscale pixels is >25, and theamount of pixels that contain local pixel variance is 35% of the totalimage area. These values were determine based on a set of trainingimages (blank, nonblank) that were fed to a trivial linear classifier.This aids in the problem of blank or near blank frames returning highmatching scores, even though they have no relevant content.

Audio Feature Extraction

FIG. 10 illustrates the general process flow and decision process forthe audio extraction. A very brief summary of the process is as follows:In step 1000, a new, incoming video file is presented to the system. Instep 1001, the audio stream of the incoming file is extracted as a 22050KHz dual channel PCM audio file. In step 1002, a decision is madedepending on the outcome of step 1001. If 5 seconds worth of audio datawas not able to be extracted, in step 1003 the process is terminatedwith no audio feature extracted. Otherwise, in step 1004 only the leftchannel is retained for analysis. In step 1005 the absolute value of theraw audio data is taken. In step 1006 the audio data is filtered with ahalf-second running mean filter. In step 1007, the data is subsampled at5 points a second. In step 1008, the featuring is complete and thefeature is built.

The following is a more detailed summary of the step-by-step process foraudio feature extraction:

-   -   1) In step 1001, a 22050 KHz PCM (WAV) dual channel audio file        is extracted from the original media file. If no WAV file is        created, or less than 5 seconds of WAY data is created, the        audio feature generation is aborted and no audio feature is        created in steps 1002 and 1003.    -   2) In step 1004, only the left channel is retained for the        feature extraction, the right channel is disregarded. For this        instantiation of the system, we only use the left channel to        avoid issues with having to mix the left and right channels        together to form one audio signal prior to feature extraction.        For another instantiation, the right channel could be used.    -   3) At step 1005, only the absolute value of the sample data is        considered. Audio signals, by nature, have amplitude values that        oscillate inversely on either side of 0. Taking the absolute        value allows the feature space to be completely positive, while        preserving the actual magnitude of the amplitude,    -   4) A running window mean filter is applied in step 1006 to the        left channel data with a size of 11025. At 22050 Khz, this        window corresponds to a half second. A half second window was        chosen as a good benchmark for the minimum average length of        silence between spoken phrases. This allows brief periods of        silence in speech to have minimal amplitude, even when mean        filtered. Mean filtering also allows audio signals to correlate        well together, even if the source audio streams differed in        dynamics or quality.    -   5) At step 1007, the resulting signal is approximated by        subsampling the data, 5 samples per second. This subsampling        greatly improves the speed of the comparison, while still        providing good signal structure, uniqueness, and discriminatory        properties. 5 samples per second were chosen for this        instantiation although other sampling rates could have been        chosen.    -   6) The final audio feature is written in binary format. As        opposed to the video features, the audio feature values are not        normalized to [0.0, 1.0]. FIG. 11 shows an example of the audio        feature signal.

Feature File System Representation

To optimize the system's processing and retrieval speed, the extractedfeatures are created and stored in binary flat files for quickdistribution, access and maintenance

Video and audio feature data is represented in one of two ways duringthe ingestion process. The first mariner is referred to as a signaturefile, which is a binary encoding for a specific feature for one specificvideo or audio stream. The second manner is referred to as a repositoryfile, which is a concatenated collection of signature files, with someadditional header information that describes the size of the repository.

Signature Files

Each feature extracted, either video or audio, is written in a binaryfile format. These “signature” files contain some metadata regarding thefile and the feature, as well as the feature itself. The signature filescontain a 58 byte header which contains metadata regarding the extractedfeature, followed by a variable amount of actual feature data. Theheader contains a 32-byte string that contains the complete md5 hashvalue for the media file. This is the primary lookup index for thisfile. Also included in the header are 2 flags, an active flag to dictatewhether or not the signature is active, and a flatline flag to identifycertain signatures as flatline (see elsewhere for more information onflatline features). The last 3 values in the signature file are valuesfor the number of features in the signature file, the number of datapoints in the data block, and the number of samples per second for thedata block. The last allows for a temporal mapping of data to time.Following the header is the actual data block. As described elsewhere,all features contain a dynamic amount of feature data that is a functionof media length. Table 1 illustrates the format of the feature signaturefiles.

TABLE 1 Description of a binary feature signature file Signature FileHeader Signature Bytes Byte Byte Bytes Bytes Bytes File Data 1-32 33 3435-42 43-50 51-58 Bytes 59- Complete char char Unsigned Unsigned DoubleDouble File MD5 value: value: 64 bit int 64 bit int float float orActive Flatline value: value: value: unsigned 0|1 0|1 Number of Numberof Data char values: features points points per Feature (NP) second Data

Repository Files

When a new audio or video file is introduced to a content managementsystem utilizing the techniques described herein, individual signaturefiles are created, and then appended to existing repositories. Arepository file has a header that describes the size of the repository,and this header is updated every time a signature is added to orremoved. In addition, to better manage the size of repository files, aswell as aid in the parallel distribution and accessing of very largerepositories, a separate repository is created for each possible hexvalue (0-F), and signatures with md5 values that begin is with aparticular hex are only inserted into the repository matching that hexvalue (via repository file name). Table 2 illustrates the format of therepository files.

TABLE 2 Description of a binary feature repository file Header, 8 byteunsigned 64 bit int value: Number of signatures Signature File 1Signature File 2 Signature File 3 . . . Signature File N

Because the header information for each signature file is intact in itscorresponding repository file, it's possible to traverse individualsignatures in a repository by calculating byte offset information fromthe signature header, and then moving to the next offset where the nextsignature starts. However, because signatures can be of varying datalengths, repository serial access is possible, but random access isimpossible unless you know the direct offsets to seek to ahead of time.

When signatures are removed from a repository file, instead ofphysically removing the signature from the file and completely rewritingthe end of the repository file, the Active flag found in the head of thesignature to be deleted is set from 1 to 0. Signatures flagged as 0 canbe quickly skipped when the complete repository is traversed during theretrieval process.

Matching Algorithm

FIG. 12 illustrates the general process flow and decision logic for thevideo matching, audio matching, and combined video and audio matchingalgorithm. A brief description of the process is as follows:

For video matching, query video features 1200 and target video features1201 are loaded. In step 1202, the question is asked if the query lengthis longer than target, and if so, the query and target are swapped(1203) for the current comparison. This is performed to simplify caseimplementation, as the shorter video is always compared to the longervideo, regardless of which was the query and which was the target. Next,if the query is a single frame (1204), then in step 1205 a specialretrieval case is invoked for single frame queries. Since crosscorrelations of single points are not statistically valid, the spatialframe features are then compared. If the query is not a single frame,then in step 1206 the question is asked whether or not the query lengthis less than 2 times the target length. If it is not, then a singlematching event is searched for (step 1207). Otherwise, possible multiplematching events are possible (step 1208). In step 1209, a crosscorrelation is performed on mean temporal feature (pass 1). If anyevent(s) have a matching score greater than or equal to the pass 1threshold (1210), then the possible matching event(s) move onto a secondpass (1212). Otherwise, the events are omitted (1211). Step 1212consists of intermittent frame comparisons, to validate the matches at aspatial frame level, and for the entire duration of the match. If theintermittent frame score is greater than the pass 2 threshold (1213),then the match is retained as a visual match with score V (1215).Otherwise, the match is omitted (1214).

For audio matching, query audio features 1218 and target audio features1219 are loaded. In step 1220, matching is performed on the features. Ifthey are not over the matching threshold in step 1221, then the targetis omitted as match in step 1222. Otherwise, the match is retained instep 1223 with audio matching score A. If the combined matching score isnot requested in step 1216, then either the video matching score 1215 orthe audio matching score 1224 is the final score. Otherwise, the productof A and V is computed in step 1225 and represents the final combinedmatching score C in step 1226.

A more detailed description of the matching algorithm follows.

The baseline algorithm for matching any temporal feature signatures is across correlation analysis. The cross correlation analysis is a standarddigital signal processing technique for finding signals embedded withinother signals with a high degree of correlation at any lag delay. Forexample, if one signal was 10 data points in length, and another 50points in length, the cross correlation would test the correlation atevery possible matching point (in this case, the smaller signal couldmatch the longer signal at any one of 40 matching points). The generalequation for the cross correlation of two arbitrary 2D signals is:

${\left( {f\; \bigstar \; g} \right)\lbrack n\rbrack}\overset{def}{=}{\sum\limits_{m = {- \infty}}^{\infty}{{f^{*}\lbrack m\rbrack}{{g\left\lbrack {n + m} \right\rbrack}.}}}$

The cross correlation is very good magnitude independent way ofcomparing signals to one another. However, one drawback to the use ofthe cross correlation analysis in this application is that very shortvideos yield very short signals (as few as 10 data points), and it'spossible for a very short signal to randomly correlate very highly to along signal, even if the original videos that the signals were extractedfrom contain no similar content. Thus, the individual spatial framefeatures are used to help in these cases, as a secondary pass.

The spatial frame features are binary in format, and encodings ofspatial information relevant to characteristics of the frame. Thefeatures are stored as collections of binary unsigned char values,aligned with each other. The comparison is a loop through all numbersrelevant to both frame spatial features, an exclusive OR operation oneach number pair, and a counting of the number of ‘1’ bits in theresult. This total, then, is divided by the total number of bitlocations possible. For example, there are 768 total bits in each framecomparison. If the total number of ‘1’ bits after all of the exclusiveOR operations was 47, then the frames are said to be 721/768 relevant,or 93.8% relevant.

Video Matching Algorithm (steps 1200-1218)

The algorithm to match video features is based on the preceding matchingalgorithm. There are several independent branches of matching logic thatare invoked, depending on properties of the query and target videosegments. Each branch will be described in detail below:

Single Frame Query Match:

Sometimes there are cases when a single query video frame is submittedfor search. Reasons for this would include: 1) That's all the usersupplied for the query, and 2) only one frame could be extracted fromthe video. In these cases, the temporal feature will only contain onepoint, and the cross correlation of single points is statisticallymeaningless. Thus, these types of searches go straight to the spatialframe features. For each frame in the target video, the frame iscompared to the query frame, and contiguous ranges of target frames thatare above the pass 2 threshold are reported as single events in theoutput matching results. The score for the event is computed as theaverage of the frame comparison matches that comprised the event.

Lengthy, Non-Flat Line Searches:

There is sufficient reason to believe that for lengthy videos whosetemporal signals contain enough variability and unique “structure”(i.e., non flat line), that the cross correlation by itself issufficient to determine accurate matches and minimize false positives.The length to determine this condition is parameterizable, but thoughempirical trials were determined to be 5 minutes (300 frames). Morespecifically stated, if the lengths of both the query and target videosegments are both >=5 minutes, and both temporal signals arenon-flatlines, then the cross correlation alone is sufficient todetermine an accurate result set. The spatial frame features are notneeded.

Single Event Matching:

If the query video length is greater than half the target video length(or vice versa), then it can logically be deduced that one and only onematching event should be searched for. In this case, a cross correlationon the temporal signals is conducted, and if an event passes the pass 1threshold, then a secondary pass (pass 2) involving an intermittentspatial frame comparison is conducted. At 20 regular intervals duringthe match (every 5% of the match duration), a spatial frame comparisonis conducted. Frames that compare with scores higher than the pass 2threshold keep the match alive. Conversely, frames that compare withscores lower than the pass 2 threshold have the capability ofpermanently cancel the match. Along with pass 2 is the notion of “framemisses”, and “frame miss tolerance”. Frame misses are the total numberof continuous frame comparisons that missed the pass 2 threshold. Framemiss tolerance is the number of continuous frame misses to toleratebefore the match is cancelled. A frame that compares higher than thepass 2 threshold will reset the number of current frame misses back tozero. If the intermittent frame comparisons reach the conclusion of thematch duration without violating the frame miss tolerance, then thematch is retained permanently with a score that is the average of theframe comparisons that were not “misses”

Multiple Event Matching:

This mode of search is very similar to the single event matchingdescribed above. Basically, one or more instances of the single searchmethod are used on completely disjoint sections of the query/targetpair. Once a positive match is found, that section is removed fromfurther consideration, and other areas are scanned for possible matches,until the entire sequence has been exhausted.

Match Output Format:

In all cases, the match output format is the same, regardless of whichmatching logic branch produced the result:

-   -   The query md5 value    -   The start and end time that the query video matched the target        video    -   The target md5 value    -   The start and end time that the target video matched the query        video    -   The final matching similarity score

Audio Matching Algorithm (steps 1218-1224)

The audio matching algorithm is simply the general matching algorithmdescribed above for one query signature against a repository ofsignatures, given an input matching similarity threshold. However, forthis match, only the correlation score is used as the matching score.Because the audio features are not normalized, the mean magnitudedifference score is not applicable. The output is a list exactly thesame as the list produced for pass 1 of the video matching algorithm.However, in the case of audio, there is only one pass. The output listof matches is final.

Combined Video and Audio Matching (Step 1216 and Those Following)

In cases where the query file has both audio and video data, and thusboth audio and video signatures, it is possible (and recommended) tocouple the audio and video retrieval matching scores together, tofurther discriminate similar videos from non similar videos. The methodto combine the audio matching score A (step 1224) with the videomatching score V (step 1215) into a combined score C (step 1226) isdefined as:

C=A×V

Several combination algorithms were examined. The first, which wasC=max(A, V), performed well at honing in on matching videos that hadeither high matching video scores or high matching audio scores.However, it suffered at accurately ranking, for example, matching videosthat had similar video quality but differing video quality. A video witha high matching video score and high matching audio score should have ahigher combined matching score than a video with a high matching videoscore and a lower matching audio score. C=max(A, V) did not allow forthis.

The second combination algorithm examined, which was C=avg(A, V),performed well at providing a general picture at how the video matchedon a combined audio and video level. However, the variance between A andV is masked. For example, if a video had a matching video score of 0.99and a matching video score of 0.91, using C=avg(A, V), this is still anadmirable combined matching score of 0.95. However, this masks the factthat the video score matched as low as 0.91, which is not ideal.

C=A×V was settled on, ultimately, because it scales the disparitybetween similar and non-similar videos geometrically, not linearly. Morespecifically, similarity scores in audio and video get amplified whenmultiplied together, further pushing similar results away fromdissimilar results. For example, if one video matched with a 0.98matching audio score and a 0.96 matching video score, using C=A×V. thecombined score would still be a high 0.94. However, if another videomatched with a 0.95 matching audio score and a 0.92 matching videoscore, using C=A×V, all of a sudden the combined score is dropped to0.87.

Retrieval Process

Four modes of searching capability are offered:

-   -   1) Video only search—only the video features of target files are        searched against    -   2) Audio only search—only the audio features of target files are        searched against    -   3) Audio or video search—On a target file by target file basis,        if both the audio and video features are present, the retrieval        uses a combined audio/video search for the asset. If not, then        the search uses either the audio or the video, whichever is        available.    -   4) Audio and video search—both the audio and video features of        target files are searched against, and targets that do not        specifically contain both are excluded from consideration.

FIG. 13 illustrates the video matching technology housed within thecompute environment for which it was designed. System disk 1301, memory1302, CPU 1303 and the video feature extraction and matching logic 1305are all interconnected within the data processing environment 1300 by abus 1304. It should be understood that other elements of a system may bepresent, such as input devices that provide video/audio data sources andoutput devices such as displays and audio speakers. However theseelements are not critical to the operation of the matching techniquesdescribed herein. The data processing environment communicates to localexternal devices via a Universal Serial Bus (USB) port 1310, andcommunicates with other networked devices via one or more NetworkInterface Cards (MC) 1306. The NIC(s) are connected directly to a LocalArea Network (LAN) router 1307. In turn, the LAN router allowsconnectivity to other LAN computes 1309 as well as the outside internet1308. The input source videos for the retrieval technique wouldoriginate from internet downloads via HTTP or FTP 1308, other networkedLAN computer disks 1309, local USB external disk 1311 or local USBoptical disk 1312, either via CD or DVD. Output retrieval results mightbe stored on local disk 1301, stored locally on external USB devices1311 1312, stored on network computer disk 1309, or served in real timeto internet-based clients 1308.

Results

Three retrieval cases are presented below, illustrating differentchallenges that the technique attempts to solve.

Case 1: Query video is a short video with no audio stream and verylittle video content variability. FIG. 14 is a screenshot of a retrievalapplication based on the technique described in this document after asearch was conducted using a short, 5 second video with very littleframe-to-frame change from a stationary camera. The search brought backall 3 other versions of the video, differing not only by file type andquality, but also frame resolution. In addition, no false positives werebrought back as matched in this result set.

Case 2: Query video is a longer video with audio. FIG. 15 is ascreenshot of the same retrieval application based on the techniquedescribed in this document. The search brought back 3 other versions ofthe video, differing not only by file type and quality, but it alsobrought back a version of the query video that was 50% the duration.This particular matching result (5-6) was a subclip of the originalquery video. In addition, no false positives were brought back asmatched in this result set.

Case 3: Query video is a longer video with audio. FIG. 16 shows ascreenshot of the same retrieval application based on the techniquedescribed in this document. The search brought back all 11 otherversions of the video, differing not only by file type and quality, butalso by letterbox and aspect ratio encoding. In addition, no falsepositives were brought back as matched in this result set.

Even though the size of these example corpuses is small (much, muchsmaller than the real-world corpuses for which the technique wasdesigned), the nature of the features being computed and thediscriminatory nature of the matching algorithm will not falter to addednoise from a larger sized repository.

Conclusion

Existing techniques utilized in academia and industry attempt to solvethe problem of video retrieval. However, many of them are based oneither 1) key frame selection and comparison, which can suffer greatlyfrom inaccuracies in the key frame selection process as well as forcinga domain-specific solution (image retrieval) on another domain (videodata), which is not scientifically optimal, and 2) a temporalrepresentation of the video, which can suffer when the content of thevideo does not yield a unique temporal signature. The preliminaryresults and evaluation indicate that the PFI technique proposed here isrobust in retrieving similar videos and is invariant to many of the realworld challenges described in the problem statement earlier in thisdocument, and is therefore novel and superior to many of the existingvideo retrieval is techniques currently employed today.

Specifically, the proposed technique is invariant to retrieving videosof different formats, as the nature of the video feature, specificallythe single point scalar measurement per frame or grid region, are robustto differences in frame encoding and quality. This also allows thetechnique to be invariant in retrieving corrupt versions of videos givennon corrupt versions, or vice versa. The proposed technique is alsorobust at accurately detecting videos containing little to no motion,regardless of video length. This is possible by use of spatial feature,which is used when two videos have been identified as having little orno content variation (flatline). The technique is multi-dimensional,meaning that it is robust at using video features when only videofeatures are available, and likewise for audio features, but willutilize both audio and video features, when present, for higheraccuracy. The technique is also robust at finding subclips within videofiles, as the cross correlation analysis attempts to find smallersignals embedded within longer signals. Lastly, the technique isinvariant to letterbox-encoded issues, due to the specific letterboxcropping filter employed See Table 3 for specifics.

TABLE 3 Listing of specific searches, and what type of challenge theyovercome. Specific Challenge Query Area Number Query Video MD5 MatchedVideo MD5 Different  7 2399288E6B2292350F20DFD10C5AADCBC799D4A31FE896825DC020537D35A1D0 formats Good  41E7496F3A18A8F941209A41605039CA6 6E5AA1131021246980C2E71A9481EC0C videoto a corrupt video Little to no 21 BAF2E6AEF34ECFB3C7F0FA4C04B04395FE9658D977FAA073D4DBD179AA828456 motion Durations 69B40286E66A9EF70C3BF1B1A62B7634CB BDD52B33E46357EDDC5A8F059D4006C5 may beslightly off Containing Many only video Containing Many only audio Clipfound 75-1 660AD92C044E5807BC05944CEC50A6AC8F9E7735BE6B4AE0739BEC4A40DC6C5A within a much larger video Changes Manyin aspect ratio or letterbox

1. A method for determining if two video signals match, the methodcomprising: extracting features from a first video signal to provide afirst feature-extracted signal; extracting features from a second videosignal to provide a second feature-extracted signal; cross-correlatingthe first and second feature-extracted signals to provide across-correlation score at each of a plurality of time lags; anddetermining a similarity score between the first and secondfeature-extracted signals based on the cross-correlation scores.
 2. Themethod of claim 1 additionally comprising: resizing the first and secondvideo signals to a common frame size prior to the feature extractorsteps.
 3. The method of claim 1 further comprising: providing anindication of a degree of match between the first and second signals. 4.The method of claim 1 wherein determining a similarity score furthercomprises: linearly combining the cross-correlation score at each lag toprovide a combined score for each lag; and determining a highestcombined score across a plurality of lags.
 5. The method of claim 4wherein linearly combining comprises weighting the cross-correlationscores equally.
 6. The method of claim 1 wherein the firstfeature-extracted signal is derived from a feature extracted from eachframe of a query video signal and the second feature-extracted signal isderived from a feature extracted from each frame of a target videosignal among a plurality of target video signals; and the method furthercomprises: determining if the target video signal is a candidate matchfor the query video signal by comparing the similarity score to athreshold; and if the target signal is a candidate match, determining ifthe target video signal matches the query video signal at an intervalcorresponding to the lag resulting in the highest combined score.
 7. Themethod of claim 6 wherein the feature is extracted from each frame ofthe is query video signal and of the target video signal based on astandard deviation of grayscale pixel values in the frame, a mean valueof grayscale pixel values in the frame, and a difference between meanvalues of respective red and blue pixels in the frame.
 8. The method ofclaim 7 wherein the feature is computed as a normalized sum.
 9. Themethod of claim 1 further comprising:: determining if each of the firstfeature-extracted signal and the second feature-extracted signal issubstantially constant over time; and if at least one of the firstfeature-extracted signal and the second feature-extracted signal is notsubstantially constant over time, determining if the first video signalmatches the second video signal based on a plurality of cell featuresextracted from respective cells forming a grid for each frame of thefirst video signal and of the second video signal, each cell featurebased on at least one of a standard deviation of grayscale pixel valuesin the corresponding cell, a mean value of grayscale pixel values in thecorresponding cell, and/or a difference between mean values ofrespective red pixels and blue pixels in the corresponding cell.
 10. Themethod of claim 9 wherein the second video signal is confirmed as amatch for the first video signal only if all of the cell features attainrespective maximal values substantially simultaneously.
 11. The methodof claim 1 further comprising: determining if each of the firstfeature-extracted signal and the second feature-extracted signal issubstantially constant over time; and if the first feature-extractedsignal and the second feature-extracted signal are substantiallyconstant over time, determining if the first video signal matches thesecond video signal based on a grayscale frequency distribution featureextracted from both the first video signal and from the second videosignal.
 12. The method of claim 1 additionally wherein the first andsecond video signals are audio-video signals, each audio-video signalcomprising audio and video data; the first and second feature-extractedsignals are derived from a video feature extracted from respective firstand second audio-video signals; and the similarity score is a videosimilarity score; and the method further comprises: cross-correlatingthird and fourth feature-extracted signals derived from audio featuresextracted from respective first and second audio-video signals toprovide an audio cross-correlation score at each of a plurality of lags;determining an audio similarity score between the third and fourthfeature-extracted signals based on the audio cross-correlation scores;multiplying the video similarity score and the audio similarity score toprovide an audio-video similarity score; and determining if the firstaudio-video signal matches the second audio-video signal by comparingthe audio-video similarity score to a threshold.